Tokenization of Tunisian Arabic: a comparison between three Machine Learning models

Tokenization represents the way of segmenting a piece text into smaller units called tokens. Since Arabic is an agglutinating language by nature, this treatment becomes crucial preprocessing step for many Natural Language Processing (NLP) applications such as morphological analysis, parsing, machine translation, information extraction, and so on. In article, we investigate word tokenization task with rewriting process to rewrite orthography stem. For task, are using Tunisian (TA) text. To best researchers’ knowledge, first study that uses TA tokenization. Therefore, start collecting preparing various corpora from different sources. Then, present comparison three character-based tokenizers based on Conditional Random Fields (CRF), Support Vector Machines (SVM) Deep Neural Networks (DNN). The proposed model CRF achieved F-measure result 88.9%.